arXiv:1505.01914vl [math.DS] 8 May 2015 


Entropic Equilibria Selection of Stationary Extrema in Einite Populations 

Marc Harpei[^ Dashiell Fryer^ 

^Department of Mathematics, Pomona College 


Abstract. We propose the entropy of random Markov trajectories originating and terminating at a state 
as a measure of the stability of a state of a Markov process. These entropies can be computed in terms of 
the entropy rates and stationary distributions of Markov processes. We apply this definition of stability to 
local maxima and minima of the stationary distribution of the Moran process with mutation and show that 
variations in population size, mutation rate, and strength of selection all affect the stability of the stationary 
extrema. 


1. Introduction 

This work is motivated by the stationary stability theorem [T], which characterizes local maxima and 
minima of the stationary distribution of the Moran process with mutation in terms of evolutionary stability. 
Specihcally, the theorem says that for sufhciently large populations, the local maxima and minima of the 
stationary distribution satisfy a selective-mutative equilibria criterion that generalizes the celebrated notion 
of evolutionary stability [2]. This means that the stationary distribution encodes the usual information 
about evolutionary stability. Precisely which equilibria are favored (i.e. are maxima or minima) is a natural 
question and depends on the choice of various parameters, such as the mutation rate the strength of 
selection /3, and the population size N. 

We propose the random trajectory entropy (RTF) of paths originating and terminating at a state as a 
measure of stability of the state m a- This is an information-theoretic quantity that is easily computable 
from the entropy rate and stationary distribution of a process, and varies continuously with the critical 
evolutionary parameters (as does the stationary distribution). We will see that RTF captures the behavior 
of the Moran process with mutation intuitively, leading to a simple method for equilibrium selection for finite 
populations, generally a significant problem in evolutionary game theory 0 i- 

2. Stationary Distributions, Fntropy Rates, and Random Trajectory Fntropies 

Our first goal is to establish the random trajectory entropy (RTF) of a state as a measure of instability of 
the state. We will be particularly concerned with the local and global extrema of the stationary distribution, 
shown in [T] to have a close connection with evolutionary stability. 

The stationary distribution of a Markov process gives the probability that the process will be in each 
state in the long run [7]. As such it is a fundamental convergence concept for Markov processes. We take 
the weighted graph viewpoint of Markov processes on a finite set of states V. Let the transition probabilities 
be given by a function T : V x V —>■ [0,1] (viewed as a matrix or a function), and the stationary distribution 
by a function s : V —>■ [0,1] (appropriately normalized to a probability distribution). We assume throughout 
that all processes are irreducible (there is a path between any two states) and have unique stationary 
distributions.Let V CV and define a stationary maximum of V to be a state v G V such that s{v') < s{v) 
for all v' G V' \ V. Then we have a local maximum v if the set V is the set of neighboring states of v and a 
global maximum if V' = V; similarly for minima. 

Although the stationary distribution of a process is often quite useful, it does not tell the full story of the 
process. While the stationary distribution gives the long run occupancy of any particular state, it does not 
explain how much the process moves among states, and so gives an incomplete description of the dynamic 
stability of a state. An entropy rate is a generalization of Shannon entropy to Markov processes and are 
commonly described as the inherent randomness or information content of a process [3]. The entropy rate 
of a process encodes both long term and short term information about the process, defined for a process X 
as follows: 

(1) H{X) = - s{vi)T{vi, Vj) logT{vi,Vj) 

-i-d 
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The entropy rate is a value attached a process rather individual states. We need a quantity associated to 
both the process and the individual states that can discriminate between equilibria. 

Following [3], define the probability of a trajectory V : r;o —t —>■•••—>■ ffc with no intermittent state 
being as the product of the transitions along the path 

Pr{V) = T{vo, vi)T{vi,V 2 ) ■ ■ ■ T{vk-i,Vk). 


Since the process is irreducible, we have that the sum over all possible such trajectories from vq to Vk is 
one, forming a probability distribution. Let T{vo,Vk) be the set of all such paths and define the random 
trajectory entropy (RTF) from vq to Vk to be the entropy of the probability distribution on T{vQ,Vk), i.e. 


Hvov;,=- Pr{v)logPriv). 

v&T(vo,Vk) 


It was shown in [3] (Theorem 1, p.l419) that when the starting and ending states are the same, the entropy 
of the random trajectory is determined by the entropy rate and the stationary probability 


H,, := H„ 


H{X) 

s{v) 


From this we have immediately have the following theorem characterizing local and global extrema of the 
stationary distribution. 


Theorem 1. For an irreducible Markov process with stationary distribution s, a state s is a local (resp. 
global) maximum (resp. minimum) if and only if the RTF Hy is a local (resp. global) minimum (resp. 
maximum). 


Furthermore we now recognize the random trajectory entropy as a measure of stationary instability of a 
state, which we can now use to compare and select equilibria for the same process and for closely related 
processes. Intuitively, a smaller RTF means that trajectories tend to stay near a local maxima, i.e that 
random walks tend to be short, which is a way of saying that the state is stable. (Note that l/s(i') is the 
expected number of steps it takes to return to v.) 


3. Applications 

We now consider several explicit examples of finite population processes. 


3.1. Moran Process -with Mutation. For the Moran process with mutation we use a special case of the 
formulation [1]; see also and [To]. Let a population be composed of n types Ai,... A„ of size N 

with Ui individuals of type Ai so that A = oi + • • • + a„. We will denote a population state by the tuple 
a = (oi,..., an) and the population distribution by a = a/N. We assume the existence of a fitness landscape 
/ where fi{d) gives the fitness of type Ap, typically /(a) = Go, for some game matrix G. (See [TT] (T^j and [T3] 
for general references on evolutionary games). Define a matrix of mutations M where 0 < Mij < 1 may be a 
function of the population state for our most general results, but we will typically assume in examples that 
for some constant value fi, the mutation matrix takes the form Mij = fj./{n — I) for i j and Mu = 1 — fi. 
A typical mutation rate is /r « 1/A. 

The Moran process with mutation is a Markov process on the population states defined by the following 
transition probabilities, corresponding to a birth-death process where birth is fitness-proportionate with 
mutation and death is uniformly random. To define the adjacent population states, let ia /3 be the vector 
that is 1 at index a, -1 at index /3, and zero otherwise, with the convention that iaa is the zero vector of 
length n. Fvery adjacent state of state a for the Moran process is of the form a + iap for some \ < a, (3 < n. 
At a population state a we choose an individual of type Ai to reproduce proportionally to its fitness, 
allowing for mutation of the new individual as given by the mutation probabilities. The distribution of 
fitness proportionate selection probabilities is given by p{a) = M(d)(p{d); explicitly, the f-th component is 


Pi{a) 


YJl=lVk{a)Mk^ 


( 2 ) 
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where the function (p{a) = aifi(a). We also randomly choose an individual to be replaced, just as in the 
Moran process. This yields the transition probabilities 

J.a+z„,p ^ for a ^ /3 

(3) t: = i- y. 

b adj a,b^a 

We will also utilize a variant incorporating a strength of selection term jS called Fermi selection [14]: 

ip{a) = 

For our examples we will restrict our attention to processes defined hy X = X(N,n, Several explicit 

examples of stationary distributions for Moran processes with mutation are given in i m- The entropy 
rate of the Moran process with mutation was computed in [15] (n = 2) and [TB] (n > 2) along with the 
development of a number of theoretical results. For our purposes the explicit values of the entropy rate are 
not needed. Generally as ^ 0, the entropy rate also goes to zero, and attains its maximum as —>■ oo for 
the neutral fitness landscape (with e.g. mutations ^ = 1/iV). The entropy rate of is bounded below by zero 
and above by log n m- The RTF is bounded below by the entropy rate, justifying the description of 
the entropy rate as the inherent randomness of a process. 

3.2. Comparison of Equilibria of a Single Process. Since the entropy rate is associated to the entire 
process, for two different states i and j we have that Hi — H{X)/s{i) and Hj — H{X)/s{j), so if H{X) ^ 0 
we need only consider the values of the stationary process in this case to compare the equilibria as Hj/Hi = 

3.2.1. Small mutation limit. Consider the special case in which the rate of mutation parameter /r —>■ 0 in 

a population of two types. Then we have lim^_>o = 0 |16| . In this case the stationary distribution 

becomes a delta distribution on the corner states. For population of two types A and B, we can express the 
limiting stationary distribution in terms of the fixation probabilities of the two types pA and pB |S]: 

lim s(0, A^) = ——— and lim s{N,Q) = ——— 

>0 Pa + pb / i — >0 Pa + pb 

Hence we have that 

lini ^ = hm ^ 

i?(7V,0) ^(O, N) PA 

In other words, the state with the type having greater fixation probability is more stable. For the classical 
Moran process with game matrix G = (^ ^) we have that (assuming r ^ 1): pA = — 'r ~^)/(1 — r~^) and 

Pb = (1 — r^“^)/(l — r~^), which gives 

lim ^ = ^ = ^ ~ 

H(Ar,o) PA l-r ^ 

As expected, whether r > 1 determines which equilibrium is favored. If r = 1, = 1/A^ = pB and the 

RTFs are equal. 

3.2.2. Large Populations and Neutral Landscapes. For arbitrarily many types, the stationary distribution for 
the neutral fitness landscape (matrix of all ones) and any mutation rate p can be analytically computed. 
For large N, the neutral landscape attains the maximum entropy rate, so for large populations a sufficient 
condition for a state for a non-neutral landscape to be more stable than the same state for the neutral 
landscape is simply to have a larger stationary probability m- For non-neutral landscapes, the large 
population limit need not maximize the entropy rate |16| . 

3.3. Comparison of Equilibria for Separate Processes on the Same States. Two instances of the 
Moran process with mutation can have the same stationary maximum state but different entropy rates. 
Consider the one-parameter family corresponding to a Hawk-Dove matrix 

(4) G=(i?), 

and transition probabilities defined by Fermi selection. For convenience fix a population size A^ > 10 and 
N even, and let the rate of mutation be p = 1/N. Then we have that {N/2, N/2) is the unique stationary 
maximum [1]. As /3 increases, the stationary probability at the maximum increases more quickly than the 
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entropy rate (which is not monotonic in this case). The net result is that the random trajectory entropy is 
decreasing as a function of /3 and that the stationary maximum is “more stable” (as would is expected for 
greater strengths of selection). See FigureFor other two player games the situation is analogous, e.g. for 
the Coordination game, the interior equilibrium RTF is decreasing as a function of f3. For both games, we 
have the intuitive result that the stability (measured by the RTF) of the extrema varies monotonically with 
the strength of selection. 

We now consider multiple examples for the landscape derived from the three-type game matrix: 


(5) 



This landscape typically has several local extrema. Let the population size N' = 6N. Then we have extrema 
at the simplex corners (6A^, 0,0), (0, 6N, 0), (0,0, 6fV), center of the boundary simplices (3A^, 3iV, 0), {3N, 0, 3N), (0, 3iV, 37V), 
and the center (2iV, 27V, 27V). Varying either fj, (Figure]^ or /3 (Figure]^ changes which equilibria have the 
smallest RTF. As /i increases, stationary probability moves from the corner points of the simplex to the 
midpoints of the boundary simplices and also toward the center. Similarly for the strength of selection /3. 

In both cases the rate of change of the stationary extrema dominates since the entropy rate varies slowly. 

Though we have focused on equilibria, the stationary distributions of finite population games can exhibit 
a variety of complex dynamical behaviors such as depicted in Figure Consider the rock-paper-scissors 
landscape given by the matrix 


/ 0 1 - 1 ' 
G = ( -1 0 1 
V 1-10- 


For some parameter choices, RPS landscapes produce an interesting stationary distribution with discretized 
cycles of constant trajectory entropy, analogous to the concentric cycles for the replicator equation and the 
fact that the relative entropy is a constant of motion of the replicator equation [12] . Assuming symmetry of 
the cycle (a large population of size divisible by 3 seems to suffice to yield approximate symmetry), no value 
on any cycle is a local maximum and the values on the maximal cycle are all global maxima. In this case 
the stationary stability theorem does not apply to the cycles (but still applies to the local minimum in the 
center of the simplex). 


3.4. Comparison of Equilibria for Process with Varying Population Size. For the final example we 
consider the effect of altering the population size the population size TV. In this case the underlying state 
spaces are different even though the equilibria are generally same) for large enough TV. For the same number 
of types n the entropy rate has the same upper bound (though the entropy rate typically increases with TV), 
and so to enable a fair comparison we normalize by the number of states (since the stationary distribution 
is spread out over a variable number of states). In general, the number of states is As for both (3 

and /i, varying the population size TV changes the favored equilibrium. See Figure We note, however, that 
the RTFs are increasing in TV, and so the issue of normalization is critical to the comparison of equilibria 
for processes with different population sizes. 


4. Discussion 

We have proposed random trajectory entropy as a measure of stability of states of finite Markov processes 
and considered several examples from finite population biology. Variations of fundamental evolutionary 
parameters alters the stability of equilibria and agrees with intuitive expectations. In particular, stability 
is closely tracked by stationary probability in several example population dynamics. We did not consider 
RTFs for paths that occur originate and terminate at different states but it is reasonable to expect that e.g. 
that a local stationary maxima will have smaller RTF in some neighborhood, and similarly for local minima. 

All computations were performed with open source code available at https; //github. com/marcharper/ 
stationary. This package can compute exact stationary distributions and entropy rates for reversible 
processes and approximate solutions for all other cases mentioned in this manuscript. All plots created with 
matplotlib m- 
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Figure 1. Right: Stationary distributions for Hawk-Dove landscapes for varying strength 
of selection /3 G [0,10], = 30, /r = 1/N. Upper Left: As /? increases, so does the stationary 

probability (blue, lower curve) of the maxima at (15,15). The entropy rate (green, upper) 
is not monotonically increasing in /3. Lower Left: Nevertheless, as j5 increases, the random 
trajectory entropy decreases monotonically as expected intuitively. More intense selection 
yields greater stability at the maximum. 
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Figure 2. This n = 3 player example for landscape defined by the matrix N = 60, 
fi = 1/A^ has multiple local stationary extrema, at the center of the simplex, on the cen¬ 
ters of the boundary simplices, and on the corners of the simplex. Left Top: The entropy 
rate of the process is given as a function of the strength of selection /3. Left Center: As 
j3 increases, the stationary probability of each extrema changes. The curves correspond as 
follows: Blue (iV, 0, 0), Green {N/2, N/2,0), Red {N/3, N/3, N/i). (Symmetric permuta¬ 
tions of these states are also extrema and have the same probabilities.) As the strength 
of selection increases, more stationary probability is concentrated on the central extrema. 
Lower Left: As /? increases, the trajectory entropy of the boundary extrema increases while 
decreasing for the central extrema, showing that strength of selection affects the stability 
of the equilibria. Which of the equilibria is most stable depends on the value of /3. Right: 
Stationary distribution for P = 0.5. 














Figure 3. Entropy Rate (Upper Left), Stationary probabilities (Center Left), and Trajec¬ 
tory entropies (Lower Left) for a process with N = 42, /3 = 1, landscape defined by Matrix 
([^ and varying rate of mutation /r. Just as for the strength of selection (3 in Figure the 
value of /i can determine which of the equilibria is most stable. As /i —>■ 0 the corner states 
are favored. As /i increases, the interior equilibrium becomes more stable. 



Figure 4. Stationary distribution for a RSP landscape (matrix below) and large population 
N = 540, 13 = 1, fj, = 3/{2N). There are apparent cycles of constant stationary probability 
and hence constant RTF. This is analogous to the concentric cycles of the replicator equation 
|13j . The central state is a local stationary maximum and has a large trajectory entropy. 
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Figure 5. Entropy Rate (Upper), Stationary probabilities (Center), and Random Trajec¬ 
tory entropies (Lower) for a process with /3 = 1, landscape defined by[^ varying population 
size N, and ^ = l/N. Just as for the strength of selection j3 in Figure]^ and /r in Figure 
the population size N can determine which of the equilibria is most stable. The trajectory 
entropies have been scaled by the number of states of the process, ■ 















